Distance calculation apparatus, imaging apparatus, and distance calculation method that include confidence calculation of distance information

ABSTRACT

A distance calculation apparatus calculates distance information based on a first image, which is generated based on luminous flux that passed through a first pupil region, and a second image, which is generated based on luminous flux that passed through a second pupil region. The distance calculation apparatus includes a distance calculation unit to calculate the distance information by comparing a local region of the first image and a local region of the second image and a confidence calculation unit to calculate confidence of the distance information based on a contrast evaluation value, which is a value indicating a magnitude of a contrast change amount in the local region of the first image, in the local region of the second image, or in a local region of a composite image of the first image and the second image.

CLAIM TO PRIORITY

This application is a divisional of copending U.S. patent applicationSer. No. 15/144,853, filed May 3, 2016, which claims the benefit ofJapanese Patent Applications No. 2015-096658, filed on May 11, 2015, andNo. 2016-019813, filed on Feb. 4, 2016, which are hereby incorporated byreference herein in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a distance calculation apparatus thatcalculates distance information based on images.

Related Art

In digital still cameras and digital video cameras, a technique ofdetecting distance information using a phase difference method is known,where pixels for distance measurement (also referred to below as“distance measurement pixels”) are disposed in a portion of or in all ofthe pixels of an image sensor.

According to this method, a plurality of photoelectric conversion unitsare disposed in a distance measurement pixel so that luminous flux,which passed through different regions on a pupil of a photographinglens, is guided to different photoelectric conversion units. Opticalimages generated by the luminous flux that passed through differentpupil regions (hereafter called “image A” and “image B”) can be acquiredby signals output by the photoelectric conversion units included in eachdistance measurement pixel, and a plurality of images can be acquiredbased on image A and image B, respectively. The pupil regioncorresponding to image A and the pupil region corresponding to image Bare decentered in different directions from each other along a so-called“pupil dividing direction” axis.

A relative positional shift according to the defocus amount is generatedbetween the plurality of acquired images (hereafter called “image A” and“image B”) along the pupil dividing direction. This positional shift iscalled an “image shift”, and the amount of the image shift is called an“image shift amount”. A distance to the object can be calculated byconverting the image shift amount into the defocus amount using apredetermined conversion coefficient. If such a method is used, highlyaccurate distance measurement can be performed at high-speed, sincethere is no need to move the lenses to measure the distance, unlike in aconventional contrast method.

To calculate the image shift amount, a region-based corresponding pointssearch technique, called “template matching”, is normally used. Intemplate matching, one of image A and image B is used as a standardimage, and the other image is used as a reference image. Further, alocal region centering around a point of interest (hereafter referred toas “collation region”) is set on the standard image, and a collationregion centering around a reference point, which corresponds to thepoint of interest, is set on the reference image. Then, whilesequentially moving the reference point, a point, at which thecorrelation (similarity) of image A and image B is highest, is searchedwithin the collation region. The image shift amount is calculated basedon the relative positional shift amount between this point and the pointof interest.

A problem that may occur in the case of calculating the image shiftamount using template matching is that calculation errors in the imageshift amount increase because of a contrast of the images. For example,if the contrast of the images is low, the change of similaritydecreases. Therefore, corresponding points cannot be correctlycalculated, and calculation errors in the image shift amount mayincrease.

To solve this problem, an apparatus disclosed in Japanese Patent No.3855812 calculates a distance to an object based on stereo images, andgenerates a distribution indicating the confidence of calculateddistances. If there is a region in which confidence is low,photographing conditions are set again so that the confidence improves,and distance is calculated again.

According to an apparatus disclosed in Japanese Patent No. 5066851, ifthere is a small region in which the defocus amount cannot becalculated, the defocus amount is interpolated for this region, so as toprevent a drop in distance measurement accuracy.

SUMMARY OF THE INVENTION

In both of the prior art arrangements mentioned above, in some cases, adesired effect may not be demonstrated depending on the conditions ofthe object.

For example, in the case of the apparatus disclosed in Japanese PatentNo. 3855812, confidence of the distance may be calculated incorrectlywhen the contrast changes only in a specific axis direction in theimages. This is because the contrast change amount is calculated basedonly on the variance values of the images included in the collationregion, without considering the pupil dividing direction.

In the case of the apparatus disclosed in Japanese Patent No. 5066851,the slope, which indicates the changes amount of the difference betweenthe images, is used, hence, noise resistance is low, and a determinationerror may occur when noise is high in the images.

In other words, the prior art arrangements have problems in terms ofimproving the distance measurement accuracy.

With the foregoing in view, it is an object of the present invention toprovide a technique to improve the accuracy of confidence for a distancemeasurement apparatus that calculates confidence in addition to thedistance information.

The present invention, in one aspect, provides a distance calculationapparatus configured to calculate distance information based on a firstimage, which is generated based on luminous flux that passed through afirst pupil region, and a second image, which is generated based onluminous flux that passed through a second pupil region, a center ofgravity of the first pupil region and a center of gravity of the secondpupil region being located at different positions along a first axis,the distance calculation apparatus comprising a distance calculationunit configured to calculate the distance information by comparing alocal region of the first image and a local region of the second image;and a confidence calculation unit configured to calculate, for aplurality of times, a first value, which indicates a contrast changeamount along the first axis, in a second axis direction crossing thefirst axis, in the local region of the first image, in the local regionof the second image or in a local region of a composite image of thefirst image and the second image, and to calculate confidence of thedistance information based on a second value, which is a valuerepresenting the plurality of first values.

The present invention, in another aspect, provides a distancecalculation apparatus configured to calculate distance information basedon a first image, which is generated based on luminous flux that passedthrough a first pupil region, and a second image, which is generatedbased on luminous flux that passed through a second pupil region, thedistance calculation apparatus comprising a distance calculation unitconfigured to calculate the distance information by comparing a localregion of the first image and a local region of the second image, and aconfidence calculation unit configured to calculate confidence of thedistance information based on a ratio between a contrast evaluationvalue, which is a value indicating a magnitude of a contrast changeamount in the local region of the first image, in the local region ofthe second image or in a local region of a composite image of the firstimage and the second image, and a noise amount estimation value which iscalculated based on the first image, the second image, or the compositeimage of the first image and the second image.

The present invention, in another aspect, provides a distancecalculation apparatus configured to calculate distance information basedon a plurality of images, which are generated based on luminous fluxthat passed through a plurality of pupil regions, comprising ageneration unit configured to generate, out of the plurality of images,a set of two images for which centers of gravity of corresponding pupilregions are located at different positions along a first axis, adistance calculation unit configured to calculate the distanceinformation by comparing the local regions that are set in the twoimages, respectively, and a confidence calculation unit configured tocalculate, for a plurality of times, a first value, which indicates acontrast change amount along the first axis, in a second axis directioncrossing the first axis, in a local region of one of the two images, orin a local region of a composite image of the two images, and tocalculate confidence of the distance information based on a secondvalue, which is a value representing the plurality of first values.

The present invention, in still another aspect, provides a distancecalculation method executed by a distance calculation apparatusconfigured to calculate distance information based on a first image,which is generated based on luminous flux that passed through a firstpupil region, and a second image, which is generated based on luminousflux that passed through a second pupil region, a center of gravity ofthe first pupil region and a center of gravity of the second pupilregion being located at different positions along a first axis, thedistance calculation method comprising the steps of calculating thedistance information by comparing a local region of the first image anda local region of the second image, and calculating, for a plurality oftimes, a first value, which indicates a contrast change amount along thefirst axis, in a second axis direction crossing the first axis, in thelocal region of the first image, in the local region of the second imageor in a local region of a composite image of the first image and thesecond image, and calculating confidence of the distance informationbased on a second value, which is a value representing the plurality offirst values.

The present invention, in yet still another aspect, provides a distancecalculation method executed by a distance calculation apparatusconfigured to calculate distance information based on a first image,which is generated based on luminous flux that passed through a firstpupil region, and a second image, which is generated based on luminousflux that passed through a second pupil region, the distance calculationmethod comprising the steps of calculating the distance information bycomparing a local region of the first image and a local region of thesecond image, and calculating confidence of the distance informationbased on a ratio between a contrast evaluation value, which is a valueindicating a magnitude of a contrast change amount in the local regionof the first image, in the local region of the second image or in alocal region of a composite image of the first image and the secondimage, and a noise amount estimation value, which is calculated based onthe first image, the second image, or the composite image of the firstimage and the second image.

The present invention, in yet still another aspect, provides a distancecalculation method executed by a distance calculation apparatusconfigured to calculate distance information based on a plurality ofimages, which are generated based on luminous flux that passed through aplurality of pupil regions, the distance calculation method comprisingthe steps of generating, out of the plurality of images, a set of twoimages for which centers of gravity of corresponding pupil regions arelocated at different positions along a first axis, calculating thedistance information by comparing the local regions which are set in thetwo images respectively, and calculating, for a plurality of times, afirst value, which indicates a contrast change amount along the firstaxis, in a second axis direction crossing the first axis, in a localregion of one of the two images, or in a local region of a compositeimage of the two images, and calculating confidence of the distanceinformation based on a second value, which is a value representing theplurality of first values.

The present invention, in still another aspect, provides anon-transitory computer readable storage medium storing a computerprogram for causing a distance calculation apparatus, configured tocalculate distance information based on a first image, which isgenerated based on luminous flux that passed through a first pupilregion, and a second image, which is generated based on luminous fluxthat passed through a second pupil region, to execute the steps ofcalculating the distance information by comparing a local region of thefirst image and a local region of the second image, and calculating, fora plurality of times, a first value, which indicates a contrast changeamount along the first axis, in a second axis direction crossing thefirst axis, in the local region of the first image, in the local regionof the second image or in a local region of a composite image of thefirst image and the second image, and calculating confidence of thedistance information based on a second value, which is a valuerepresenting the plurality of first values, wherein a center of gravityof the first pupil region and a center of gravity of the second pupilregion are located at different positions along a first axis.

The present invention, in yet still another aspect, provides anon-transitory computer readable storage medium storing a computerprogram for causing a distance calculation apparatus, configured tocalculate distance information based on a first image, which isgenerated based on luminous flux that passed through a first pupilregion, and a second image, which is generated based on luminous fluxthat passed through a second pupil region, to execute the steps ofcalculating the distance information by comparing a local region of thefirst image and a local region of the second image, and calculatingconfidence of the distance information based on a ratio between acontrast evaluation value, which is a value indicating a magnitude of acontrast change amount in the local region of the first image, in thelocal region of the second image or in a local region of a compositeimage of the first image and the second image, and a noise amountestimation value, which is calculated based on the first image, thesecond image, or the composite image of the first image and the secondimage.

The present invention, in yet still another aspect, provides anon-transitory computer readable storage medium storing a computerprogram for causing a distance calculation apparatus, configured tocalculate distance information based on a plurality of images, which aregenerated based on luminous flux that passed through a plurality ofpupil regions, to execute the steps of generating, out of the pluralityof images, a set of two images for which centers of gravity ofcorresponding pupil regions are located at different positions along afirst axis, calculating the distance information by comparing the localregions which are set in the two images respectively and calculating,for a plurality of times, a first value, which indicates a contrastchange amount along the first axis, in a second axis direction crossingthe first axis, in a local region of one of the two images, or in alocal region of a composite image of the two images, and calculatingconfidence of the distance information based on a second value, which isa value representing the plurality of first values.

According to the present invention, accuracy of confidence can beimproved in a distance calculation apparatus that calculates confidence,in addition to the distance information.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A to FIG. 1C are diagrams depicting configurations of a digitalcamera and an image sensor according to Embodiment 1.

FIG. 2A and FIG. 2B are diagrams depicting the relationships between aphotoelectric conversion unit and a pupil region, and between an exitpupil and a pupil region.

FIG. 3A to FIG. 3C are processing flow charts of a digital cameraaccording to Embodiment 1.

FIG. 4A and FIG. 4B are diagrams depicting steps of calculating afocusing distance.

FIG. 5A to FIG. 5D are first diagrams depicting a contrast change and animage shift amount.

FIG. 6A to FIG. 6C are second diagrams depicting a contrast change andan image shift amount.

FIG. 7A and FIG. 7B are processing flow charts of a digital cameraaccording to Embodiment 2.

FIG. 8A and FIG. 8B are diagrams depicting an image sensor and an exitpupil according to Embodiment 3.

FIG. 9A and FIG. 9B are processing flow charts of a digital cameraaccording to Embodiment 3.

FIG. 10A and FIG. 10B are processing flow charts of a digital cameraaccording to Embodiment 4.

FIG. 11A and FIG. 11B are graphs for describing a noise approximationmethod according to Embodiment 4.

FIG. 12A and FIG. 12B are processing flow charts of a digital cameraaccording to Embodiment 5.

FIG. 13 is a processing flow chart of a digital camera according to anembodiment.

FIG. 14A and FIG. 14B are processing flow charts of a digital cameraaccording to Embodiment 6.

FIG. 15A and FIG. 15B are diagrams depicting a configuration of adigital camera according to a modification.

DESCRIPTION OF THE EMBODIMENTS Embodiment 1

Embodiment 1 of the present invention will now be described withreference to the drawings.

Embodiment 1 is an imaging apparatus (digital camera) equipped with adistance calculation apparatus according to the present invention, butthe present invention is not limited to this. In the present invention,the same constituent elements are denoted with the same referencesymbols, and a redundant description is omitted.

FIG. 1A is a diagram depicting a configuration of a digital camera 100according to Embodiment 1.

The digital camera 100 includes an imaging optical system 120, an imagesensor 101, a distance calculation unit 102, an image storage unit 104,an image generation unit (not illustrated), and a lens driving controlunit (not illustrated). Of these elements, the imaging optical system120, the image sensor 101, the distance calculation unit 102, and theimage storage unit 104 constitute a distance calculation apparatus 110.

The imaging optical system 120 is a photographing lens of the digitalcamera 100, and is a unit for forming an image of an object on the imagesensor 101, which is an image detector plane. The imaging optical system120 is constituted by a plurality of lens groups and a diaphragm, andforms an exit pupil 130 at a position distant from the image sensor 101by a predetermined distance. The reference number 140 in FIG. 1Aindicates an optical axis of the imaging optical system 120, and is anaxis parallel with the Z axis in this embodiment. The X axis and the Yaxis are perpendicular to each other and to the optical axis.

The image sensor 101 is constituted by a CMOS (Complementary Metal OxideSemiconductor) or a CCD (Charge Coupled Device). An object image formedon the image sensor 101 by the imaging optical system 120 is convertedinto electrical signals by the image sensor 101. The image sensor 101will now be described in detail with reference to FIG. 1B.

FIG. 1B is an X-Y cross-sectional view of the image sensor 101. In theimage sensor 101, a plurality of pixel groups 150 (each pixel groupconsists of 2×2 pixels) are arrayed. In each pixel group 150, greenpixels 150G1 and 150G2 are diagonally disposed, and a red pixel 150R anda blue pixel 150B are disposed in the other two pixels.

The green pixels 150G1 and 150G2, the red pixel 150R and the blue pixel150B each has a plurality of photoelectric conversion units.

FIG. 1C is a schematic diagram of the I-I′ cross section of the pixelgroup 150. Each pixel is constituted by a light receiving layer 182 anda light guiding layer 181. In the light receiving layer 182, twophotoelectric conversion units (first photoelectric conversion unit 161and second photoelectric conversion unit 162), for converting receivedlight into electrical signals, are disposed. In the light guiding layer181, on the other hand, a micro-lens 170 for efficiently guiding theluminous flux that entered the pixel to the photoelectric conversionunit, a color filter (not illustrated), which passes light having apredetermined wavelength band, wires (not illustrated) for reading theimage and driving the pixel, and the like, are disposed.

In this embodiment, it is assumed that the image sensor 101 outputsdigitally converted electrical signals. Hereafter, a set of digitalsignals corresponding to one image is simply referred to as an “image”.The image is RAW format data, and is developed by an image generationunit (not illustrated) that is viewable by the user.

The distance calculation unit 102 is a unit for calculating distanceinformation based on the image generated by the image generation unit.Detailed processing content will be described later. Distanceinformation refers to information on the relative position of theobject, such as a distance to the object (hereafter referred to as“focusing distance”), a defocus amount, and an image shift amount, andmay also be called “depth information”. The distance calculation unit102 may be implemented by specially designed hardware or may beimplemented by a software module. The distance calculation unit 102 mayalso be implemented by a field programmable gate array (FPGA), anapplication specific integrated circuit (ASIC), or the like, or may beimplemented by a combination of these components.

If the above mentioned units are constructed by software, a programstored in an auxiliary storage device is loaded to the main storagedevice, and the program is executed by the CPU, whereby each unit canfunction (the CPU, the auxiliary storage device, and the main storagedevice are not illustrated).

Now, a processing when the digital camera 100 according to thisembodiment photographs an image will be described with reference to theprocessing flow chart in FIG. 13. The processing in FIG. 13 is executedby a controller (not illustrated) while the shutter button of thedigital camera 100 is being depressed (including half depression).

First, in step S1301, lens information (e.g., focal length, diaphragmstop) is read from the imaging optical system 120 and temporarilystored.

The steps S1302 to S1304 are processings to perform focusing on anobject.

First, in step S1302, defocus amount and confidence of the calculateddistance (hereafter referred to as “distance confidence”) are calculatedby the later mentioned distance calculation processing, based on theimages output by the image sensor 101.

Then, in step S1303, it is determined whether the imaging optical system120 is in the focused state or not, based on the calculated defocusamount and distance confidence. If the target object is not focused on,processing advances to step S1304 where the lens driving control unit iscontrolled based on the calculated defocus amount, and the imagingoptical system 120 is driven to the focusing position.

If it is determined that the target object is focused on in step S1303,processing advances to step S1305, where it is determined whether theshutter was released (full depression). If it is determined that theshutter was not released, then processing returns to step S1302, and theabove mentioned processing is repeated.

If it is determined that the shutter was released in step S1305, theimage is acquired from the image sensor 101 and stored in the imagestorage unit 104.

The digital camera 100 according to this embodiment can generate anornamental image by the image generation unit performing developmentprocessing on an image stored in the image storage unit 104. Further, ifthe distance calculation processing similar to that executed in stepS1302 is performed on the image stored in the image storage unit 104, adistance image corresponding to the ornamental image and distribution ofconfidence corresponding to this distance image can be generated.

Now, a method for calculating a focusing distance based on images outputby the image sensor 101 will be described.

FIG. 2A is a diagram depicting luminous flux received by the firstphotoelectric conversion unit 161 and the second photoelectricconversion unit 162 of the image sensor 101. FIG. 2A shows only the exitpupil 130 of the imaging optical system 120 and a representative pixel(green pixel 150G1) disposed in the image sensor 101.

The micro-lens 170 in the pixel 150G1, shown in FIG. 2A and FIG. 2B, isdisposed so that the exit pupil 130 and the light receiving layer 182are optically conjugate with each other. As a result, the luminous fluxthat passed through a first pupil region 210 included in the exit pupil130 enters the first photoelectric conversion unit 161. In the samemanner, the luminous flux that passed through a second pupil region 220enters the second photoelectric conversion unit 162.

The plurality of first photoelectric conversion units 161 disposed ineach pixel photoelectrically convert the received luminous flux, andgenerate a first image. In the same manner, the plurality of secondphotoelectric conversion units 162 disposed in each pixelphotoelectrically convert the received luminous flux, and generate asecond image. Hereafter, the first image is also called “image A”, andthe second image is also called “image B”.

From the first image, intensity distribution of an image, which luminousflux passed mainly through the first pupil region formed on the imagesensor 101, can be acquired, and from the second image, intensitydistribution of an image, which luminous flux passed mainly through thesecond pupil region formed on the image sensor 101, can be acquired.Hereafter, the former is also called “image A”, and the latter is alsocalled “image B”.

FIG. 2B is a diagram of the exit pupil 130 of the imaging optical system120 viewed from the intersection of the optical axis 140 and the imagesensor 101. Here, a center of gravity position of the first pupil region210 is indicated by a first center of gravity position 211, and a centerof gravity position of the second pupil region 220 is indicated by asecond center of gravity position 221.

In this embodiment, the first center of gravity position 211 isdecentered (shifted) from the center of the exit pupil 130 along a firstaxis 200. The second center of gravity position 221, on the other hand,is decentered (shifted) from the center of exit pupil 130 along thefirst axis 200 in the opposite direction of the shifting direction ofthe first center of gravity position 211. In other words, the firstpupil region and the second pupil region are decentered along the firstaxis in different directions from each other.

In this embodiment, the first photoelectric conversion unit 161 isshifted from the center point of the green pixel 150G1 along the X axis(first axis) in the negative direction. In other words, the first centerof gravity position 211 is decentered along the first axis in thepositive direction.

The second photoelectric conversion unit 162, on the other hand, isshifted from the center point of the green pixel 150G1 along the X axisin the positive direction. In other words, the second center of gravityposition 221 is decentered along the first axis in the negativedirection.

The relative positional shift amount between the first image and thesecond image becomes the image shift amount between image A and image B.The image shift amount is an amount corresponding to the defocus amount.Therefore, the distance to the object can be calculated by calculatingthe image shift amount between the first image and the second image by alater mentioned method, and converting this image shift amount into thedefocus amount using a conversion coefficient.

In FIG. 2A and FIG. 2B, the first pupil region is illustrated assumingthat X is positive, and the second pupil region is illustrated assumingthat X is negative, but, in reality, the light that reaches the lightreceiving layer 182 has a certain degree of spread due to thediffraction phenomenon of light. A certain degree of spread is alsogenerated by the crosstalk of carriers inside the light receiving layer182.

In other words, on this occasion luminous flux that passed through aregion of which X is negative enters the first photoelectric conversionunit 161, and luminous flux that passed through a region of which X ispositive enters the second photoelectric conversion unit 162. This meansthat the first pupil region and the second pupil region are not clearlyseparated, but have overlapping regions. In the description of thisembodiment, however, it is assumed for convenience that the first pupilregion and the second pupil region are clearly separated.

Now, the processing to calculate the distance to the object using thefirst image and the second image will be described with reference toFIG. 3A, which is a processing flow chart performed by the distancecalculation unit 102.

First, in step S1, the first image and the second image, which wereacquired by the image sensor 101 and stored in the image storage unit104, are acquired.

Then, in step S2, processing to correct the light quantity balancebetween the first image and the second image is performed. For themethod of correcting the light quantity balance, a known method can beused. For example, images generated by photographing a uniform surfacelight source are stored in advance, and a coefficient for correcting thelight quantity balance is calculated using these images.

Step S3 is a step of calculating a distance to the object based on thefirst image and the second image (distance calculation step). Thedistance calculation step will be described with reference to FIG. 3Band FIG. 4A.

FIG. 3B is a flow chart depicting the processing performed in step S3 indetail.

First, in step S31, an image shift amount of the second image withrespect to the first image is calculated. The method of calculating theimage shift amount will be described with reference to FIG. 4A.

In FIG. 4A, reference number 401 indicates the first image, andreference number 402 indicates the second image. It is assumed that theX axis is the first axis.

In step S31, the first image 401 is selected as a standard image, and acollation region 420 is set on the first image 401. A collation regionis a local region centering around a point of interest 410. If thecollation region 420 is small, a calculation error is generated in theimage shift amount due to local operation. Hence, it is preferable thatthe size of the collation region is about 9 pixels×9 pixels.

Then, the second image 402 is selected as a reference image, and acollation region 421 is set in the second image 402 in the same manner.The collation region that is set in the second image is a local regioncentering around the reference point 411.

Then, while moving the reference point 411 along the first axis (Xaxis), the correlation between the image in the collation region 420(first image) and the image in the collation region 421 (second image)is calculated, and a reference point, at which correlation is highest,is determined to be the corresponding point. A relative positional shiftamount between the point of interest 410 and the corresponding point isdetermined, and this is defined as the image shift amount.

By performing this processing while sequentially moving the point ofinterest 410 along the first axis, the image shift amount at each pixelposition in the first image can be calculated. For the method ofcalculating the correlation, a known method can be used. For example, amethod called SSD (Sum of Squared Difference), where a sum of squares ofthe differences between pixel values is used as an evaluation value, canbe used.

Then, in step S32, the image shift amount acquired in step S31 isconverted into the defocus amount using a predetermined conversioncoefficient. The defocus amount is a distance from the image sensor 101to the imaging plane, where an image is formed by the imaging opticalsystem 120.

Here, the defocus amount ΔL can be given by Expression (1), where ddenotes an image shift amount, w denotes a baseline length which is aconversion coefficient, and L denotes a distance from the image sensor101 to the exit pupil 130.

[Math.  1] $\begin{matrix}{{\Delta \; L} = \frac{d \cdot L}{w - d}} & {{Expression}\mspace{14mu} (1)}\end{matrix}$

The baseline length w is a distance between the first center of gravityposition 211 and the second center of gravity position 221 shown in FIG.2B. In this embodiment, the image shift amount is converted into thedefocus amount using Expression (1), but in Expression (1), it isapproximated that w is much greater than d (w>>d). Hence, the followingExpression (2) may be used.

[Math. 2]

ΔL=Gain·d  Expression (2)

In step S33, the defocus amount calculated in step S32 is converted intothe focusing distance. The conversion from the defocus amount to thefocusing distance can be performed using the imaging relationship of theimaging optical system 120.

By the processing in step S3, distribution of focusing distance(hereafter referred to as “distance image”) of the acquired image can beacquired.

Refer back to FIG. 3A to continue the description.

Step S4 is a step of calculating distribution of confidence of thefocusing distances (hereafter referred to as “confidence image”)calculated in step S3 (confidence calculation step). The confidencecalculation step will be described with reference to FIG. 3C and FIG.4B. FIG. 3C is a detailed flow chart depicting the step performed instep S4.

First, in step S41, the contrast change amount is calculated along thefirst axis.

The method of calculating the contrast change amount will be describedwith reference to FIG. 4B. FIG. 4B is a diagram depicting the firstimage 401, and the point of interest 410 and the collation region 420,which are set in this image.

In FIG. 4B, a hatched region 430 indicates one pixel. In this example,it is assumed that the collation region 420 is in a range where the Xcoordinate is Xp to Xq and the Y coordinate is Yp to Yq. It is alsoassumed that the coordinates of the point of interest 410 is (Xc, Yc).

In this embodiment, as the contrast change amount, the variance of eachpixel value of the first image included in the collation region 420 iscalculated for each Y coordinate along the first axis (that is, X axis).

The contrast change amount C(x,y) can be given by Expression (3) andExpression (4). I(x,y) is a pixel value of the first image at a position(x,y) in a pixel array, and Nx is a number of pixels in the X axisdirection included in the collation region 420.

Hereafter, the contrast change amount acquired for each Y coordinate isreferred to as “first value”.

[Math.  3] $\begin{matrix}{{C\left( {x,y} \right)} = {\frac{1}{Nx}{\sum\limits_{{xi} = {xp}}^{xq}\; {\left( {{I\left( {{xi},y} \right)} - {Iave}} \right)^{2}\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack}}}} & {{Expression}\mspace{14mu} (3)} \\{{Ivae} = {\frac{1}{Nx}{\sum\limits_{{xi} = {xp}}^{xq}\; {I\left( {{xi},y} \right)}}}} & {{Expression}\mspace{14mu} (4)}\end{matrix}$

Then, in step S42, a representative value of a plurality of first valuesacquired in the second axis (that is, Y axis) direction (hereafter, thisrepresentative value is referred to as “second value”). In thisembodiment, a mean value of the first values is calculated as therepresentative value of the first values.

In other words, the representative value Conf(x,y) of the contrastchange at the point of interest 410 is given by Expression (5). Nydenotes a number of pixels in the Y direction included in the collationregion 420.

[Math.  5] $\begin{matrix}{{{Conf}\left( {x,y} \right)} = {\frac{1}{Ny}{\sum\limits_{{yi} = {yp}}^{yq}\; {C\left( {x,{yi}} \right)}}}} & {{Expression}\mspace{14mu} (5)}\end{matrix}$

In this embodiment, a set of the representative values of the contrastchange (Conf) determined like this is defined as a distance confidencecorresponding to the focusing distance.

Here, as the variance of the first value is greater, the contrast changein the first image in the first axis direction is greater. Further, themagnitude of the contrast change amount that contributes to thecalculation of the image shift amount of the first image, included inthe collation region 420, can be calculated by averaging the variancesof the pixel values along the second axis.

In the distance calculation apparatus 110 according to this embodiment,the first photoelectric conversion unit 161 and the second photoelectricconversion unit 162 are disposed side-by-side along the X axis, which isthe first axis. Therefore, an image shift is generated between the firstimage and the second image along the first axis.

In such a case, the image shift amount can be accurately detected if theobject has the contrast change in a direction parallel with the X axis,but the detection error in the image shift amount increases if thecontrast change of the object is small in a direction parallel with theX axis.

In other words, an error in the image shift amount can be estimatedbased on the magnitude of the contrast change.

The relationship between the contrast change and the image shift amountwill be described with reference to FIG. 5A to FIG. 5D.

FIG. 5A shows the brightness distribution of an object 510 having acontrast change in the direction parallel with the X axis, and the firstimage 511 and the second image 512, which are acquired by imaging thisobject.

The solid line in FIG. 5B indicates the pixel values of the first image511, and the broken line indicates the pixel values of the second image.As this graph shows, the first image 511 and the second image 512 areshifted in the X direction if the object 510 is in the defocus state.

The object 510 has a contrast change in the brightness values in the Xdirection Therefore, the image shift amount can be accurately calculatedby using the corresponding point search method, which was described withreference to FIG. 4A.

FIG. 5C, on the other hand, shows the brightness distribution of anobject 520 having a contrast change in the direction parallel with the Yaxis, and the first image 521 and the second image 522, which areacquired by imaging this object.

Just like FIG. 5B, FIG. 5D shows the pixel values of the first image inthe X direction and the pixel values of the second image in the Xdirection, but in FIG. 5D, the solid line and the broken line overlap.This is because an image shift was generated between the first image andthe second image in the X direction. Therefore, if there is no contrastchange in the X direction like this, it is difficult to detect the imageshift amount.

Therefore, in this embodiment, the contrast change amount (first value)is calculated first by calculating the variances of the pixel valuesalong the first axis (X axis) in step S41, and then, the mean value ofthe first values (second value) is calculated along the second axis instep S42. The second value acquired like this is defined as theconfidence of the distance.

If the distance confidence is calculated by this method, the evaluationresult indicating high confidence is acquired for the object 510 in FIG.5A, and the evaluation result indicating low confidence is acquired forthe object 520 in FIG. 5C. In other words, the distance confidence canbe correctly calculated.

In the case of the confidence calculation method disclosed in JapanesePatent No. 3855812, on the other hand, the variances of all the pixelvalues included in the collation region are calculated withoutconsidering the direction in which the image shift is generated. Inother words, the variances become approximately the same values in bothFIG. 5A and FIG. 5C. Therefore, if an object having a contrast changeonly in the Y direction is imaged, as in the case of FIG. 5C, anincorrect distance confidence may be generated.

Some of the prior art techniques for calculating confidence usecorrelation. For example, in the case of the apparatus disclosed inJapanese Patent No. 5066851, confidence is calculated by using the ratiobetween the minimum value of the correlation value and the slopecalculated from the correlation value near the minimum value. In such acase, if local noise exists, the correlation value disperses, and as aresult, a larger slope may be calculated or smaller minimum value may becalculated in error.

FIG. 6A shows the brightness distribution of an object 610 of whichcontrast ratio is 2.0, and a first image 611 and a second image 612,which were acquired by imaging this object. FIG. 6B shows the brightnessdistribution of an object 620 of which contrast ratio is 1.25, and afirst image 621 and a second image 622, which were acquired by imagingthis object. In both FIG. 6A and FIG. 6B, noise corresponding to ISOsensitivity 100 has been added.

FIG. 6C shows graphs on the object 610 and the object 620, respectively,where the abscissa indicates the moving distance of the reference point,and the ordinate indicates the correlation value using SSD (that is, thesum of squares of the differences of the pixel values). The solid line613 in FIG. 6C indicates the correlation value of the object 610, andthe broken line 623 indicates the correlation value of the object 620.Since SSD is used here, the correlation is higher as the correlationvalue is smaller. The arrow mark 630 indicates the correct value of theimage shift amount, and the arrow mark 640 indicates the image shiftamount calculation result of the object 610. The arrow mark 650indicates the image shift amount calculation result of the object 620.As FIG. 6C shows, the image shift amount can be more accuratelycalculated as the contrast ratio is higher.

Table 1 shows the comparison of the confidence of distance calculated inthe confidence calculation step according to this embodiment, and theconfidence calculated using the ratio of the minimum value of thecorrelation value and the slope of the correlation value near theminimum value, as disclosed in Japanese Patent No. 5066851 (hereafterreferred to as “comparative example”).

The distance confidence according to this embodiment is higher as thevalue thereof is greater, and the distance confidence according to thecomparative example is higher as the value thereof is smaller. In thecase of the comparative example, high distance confidence was calculatedfor the object 620, even if the calculation error of the image shiftamount is large. This is because, in an area near the minimum value ofthe correlation values, a larger slope of the correlation value near theminimum value is evaluated due to the influence of local noise.

In the case of the method of calculating distance confidence accordingto this embodiment, on the other hand, the contrast change of the pixelvalues included in the collation region in the X direction is evaluatedby variance. Therefore, evaluation is hardly influenced by local noise.As a result, if an object of which contrast ratio is low, such as thecase of the object 620, is imaged, a low distance confidence iscalculated.

TABLE 1 Distance Distance confidence of Correct confidence ofcomparative distance this embodiment example confidence Object 610 High(evaluation Low (evaluation High (calculation error value = 70) value =33.7) of image shift amount is small) Object 620 Low (evaluation High(evaluation Low (calculation error value = 52) value = 6.7) of imageshift amount is large)

According to this embodiment, the distance is calculated first in stepS3, and then, the distance confidence is calculated in step S4, but stepS3 and step S4 may be executed in reverse. By executing step S4 beforestep S3, the distance confidence can be acquired in advance. Thenparameters used for calculating the focusing distance (e.g., size ofcollation region 420, range of searching corresponding points) can beset based on the distance confidence acquired in advance, wherebycalculation accuracy of the focusing distance can be improved.

As described above, in the case of the distance calculation apparatusaccording to this embodiment, the contrast change amount is calculatedalong the first axis, which is the pupil dividing direction axis, bycalculating variances of the pixel values along the first axis. Further,if the representative value of the contrast change amount is calculatedalong the second axis, only the magnitude of the contrast change in thefirst axis direction included in the collation region can be evaluated.

As a result, only the contrast change that contributes to thecalculation of the image shift amount can be evaluated, and the distanceconfidence can be accurately calculated. Further, the influence of localnoise can be reduced by performing statistical evaluation using thepixels included in the collation region.

In this embodiment, the variances of the pixel values are calculated asthe contrast change amount, but values other than the variances may beused as long as the magnitude of the contrast change can be evaluated.For example, a standard deviation, an absolute value of the differencebetween the maximum value and the minimum value of the pixel values, orthe maximum value (or a mean value) of the absolute values when thepixel values are differentiated in the first axis direction may be used.To minimize the influence of noise, it is preferable to use thevariances or the standard deviation, which are the result of statisticalevaluations.

In this embodiment, the mean value of the first values was used as thesecond value, but a value other than a mean value may be used as long asthe representative value of the first values in the collation region canbe calculated. For example, a total value, a maximum value, a minimumvalue, a median, a mode, or the like, may be used.

The method of calculating the second value, however, preferablycorresponds to the correlation value computing method which was used forcalculating the image shift amount. In other words, if the sum ofabsolute values of the differences or the sum of squares of thedifferences is used as the correlation value of the image shift amountcomputation. Then, it is preferable to use the total value or the meanvalue, which is calculated based on the sum.

In this embodiment, the contrast change amount is calculated using onlythe first image in step S4, but both the first image and the secondimage may be used. Further, a third image may be generated by combiningthe first image and the second image, and this third image may be usedfor calculating the contrast change amount. In this case, the firstimage and the second image can be combined using the sum of or the meanvalue of the pixel values of the corresponding pixel positions.

Embodiment 2

In Embodiment 1, the distance confidence calculation step (step S4) isexecuted after executing the distance calculation step (step S3). InEmbodiment 2, on the other hand, the distance calculation and thedistance confidence calculation are executed in a same step.

FIG. 7A is a processing flow chart performed by the distance calculationunit 102 according to Embodiment 2.

A difference between Embodiment 2 and Embodiment 1 is that the focusingdistance and the distance confidence are calculated simultaneously instep S5 of Embodiment 2, instead of step S3 and step S4 of Embodiment 1.

Now, the processing to be executed in step S5 will be described indetail with reference to FIG. 7B.

A description of step S31, which is the same as Embodiment 1, will beomitted.

In this embodiment, after the image shift amount is calculated in stepS31, a step of generating a third image by combining the first image andthe second image, and calculating the contrast change amount using thethird image (step S51) is executed.

In concrete terms, the point of interest and the collation region usedfor calculating the image shift amount in step S31 are set for the firstimage. Then, the corresponding point calculated in step S31 and acollation region centering around the corresponding point are set forthe second image. Then, a third image is generated, which is a compositeimage of the first image and the second image included in the collationregion. The composite image can be generated using the sum of or themean of the corresponding pixels.

Then, using the generated third image, variances of the third image arecalculated along the first axis, just like step S41, whereby the firstvalue is calculated.

A description of step S42, step S32, and step S33, which are the same asEmbodiment 1, will be omitted.

In Embodiment 2, the contrast change amount is calculated like this,using the third image, which is an image generated by combining thefirst image and the second image. Since the contrast change amount canbe calculated under conditions similar to those used when the imageshift amount was calculated, the calculation accuracy of the distanceconfidence can be improved.

In some cases, the contrast change amount may be different between thefirst image and the second image depending on the imaging environment,such as when vignetting of the imaging optical system is generated inthe peripheral angle of view of the image sensor 101. In Embodiment 2,however, the contrast change amount is calculated after integrating thefirst image and the second image. Hence, even in such a case,calculation accuracy of the distance confidence can be ensured.

In this embodiment, the third image is generated after calculating theimage shift amount in step S31, but the third image may be generatedbefore calculating the image shift amount. In concrete terms, the thirdimage may be generated by combining the image of the collation region,which was used for calculating the image shift amount, of the firstimage (signals of a plurality of first photoelectric conversion units)and a corresponding image of the collation region of the second image(signals of a plurality of second photoelectric conversion units formingpairs with the plurality of first photoelectric conversion units). Ifthe image sensor 101 is configured to output signals generated bycombining the signals of the first photoelectric conversion units 161and the signals of the second photoelectric conversion units 162, it ispreferable to generate the third image in this way.

Embodiment 3

In Embodiments 1 and 2, two photoelectric conversion units are used foreach pixel. In Embodiment 3, however, four photoelectric conversionunits are disposed in each pixel.

FIG. 8A is an X-Y cross-sectional view of an image sensor 801 accordingto Embodiment 3. In the image sensor 801 according to Embodiment 3, aplurality of pixel groups 850 (each pixel group consists of 2×2 pixels)are arrayed. In concrete terms, green pixels 850G1 and 850G2 arediagonally disposed, and a red pixel 850R and a blue pixel 850B aredisposed in the other two pixels.

In each pixel, four photoelectric conversion units (first photoelectricconversion unit 861, second photoelectric conversion uni 862, thirdphotoelectric conversion unit 863, and fourth photoelectric conversionuni 864) are juxtaposed.

FIG. 8B is a diagram of the exit pupil 130 of the imaging optical system120 viewed from the intersection of the optical axis 140 and the imagesensor 801. A plurality of first photoelectric conversion units disposedin each pixel photoelectrically-convert the luminous flux that passedthrough a first pupil region 810 and generate a first image. In the sameway, the second photoelectric conversion unit, the third photoelectricconversion unit and the fourth photoelectric conversion unitphotoelectrically-convert the luminous flux that passed through a secondpupil region 820, a third pupil region 830, and a fourth pupil region840, and generate a second image, a third image and a fourth image,respectively.

In FIG. 8B, a center of gravity position of the first pupil region 810is indicated by a first center of gravity position 811, and a center ofgravity position of the second pupil region 820 is indicated by a secondcenter of gravity position 821. Further, a center of gravity position ofthe third pupil region 830 is indicated by a third center of gravityposition 831, and a center of gravity position of the fourth pupilregion 840 is indicated by a fourth center of gravity position 841.

Furthermore, a center of gravity position of a combined region(hereafter referred to as “total region”) of the first pupil region andthe fourth pupil region is indicated by a fifth center of gravityposition 851, and a center of gravity position of a total region of thesecond pupil region and the third pupil region is indicated by a sixthcenter of gravity position 861. Further, a center of gravity position ofa total region of the first pupil region and the second pupil region isindicated by a seventh center of gravity position 871, and a center ofgravity position of a total region of the third pupil region and thefourth pupil region is indicated by an eighth center of gravity position881.

In Embodiment 3, the direction of generating the image shift can bechanged by changing the combination of images to be combined.

For example, in a fifth image, which is an image generated by combiningthe first image and the fourth image, the luminous flux that passedthrough the first pupil region 810 and the fourth pupil region 840generates intensity distribution of an image formed on the image sensor801. In a sixth image, which is an image generated by combining thesecond image and the third image, the luminous flux that passed throughthe second pupil region 820 and the third pupil region 830 generatesintensity distribution of an image formed on the image sensor 801.

The center of gravity position 851 of the total region corresponding tothe fifth image and the center of gravity position 861 of the totalregion corresponding to the sixth image are decentered along the X axisin different directions from each other. Therefore, the image shift isgenerated along the X axis. In this case, the above mentioned distanceconfidence calculation can be performed by defining the X axis as thefirst axis and the Y axis as the second axis.

Now, a method of generating the image shift along the Y axis will bedescribed.

In a seventh image generated by combining the first image and the secondimage, the luminous flux that passed through the first pupil region 810and the second pupil region 820 generates intensity distribution of animage formed on the image sensor 801. In an eighth image generated bycombining the third image and the fourth image, the luminous flux thatpassed through the third pupil region 830 and the fourth pupil region840 generates intensity distribution of an image formed on the imagesensor 801.

The center of gravity position 871 of the total region corresponding tothe seventh image and the center of gravity position 881 of the totalregion corresponding to the eighth image are decentered along the Y axisin different directions from each other. Therefore, the image shift isgenerated along the Y axis. In this case, the above mentioned distanceconfidence calculation can be performed by defining the Y axis as thefirst axis and the X axis as the second axis.

By changing the combination of images to be combined in this way, theaxis along which the image shift is generated can be changed.

In Embodiment 3, the distance confidence is calculated by appropriatelysetting the first axis and the second axis according to the axis alongwhich the image shift is generated. Thereby, the axis along which theimage shift is generated can be changed according to the direction ofthe contrast of the object, and the distance to the object can beaccurately calculated.

For example, if an object has a contrast change only in the X direction,as in the case of object 510 in FIG. 5A, the image shift amount iscalculated using the fifth image and the sixth image. Since the imageshift is generated along the X axis in this case, the distance to theobject can be accurately calculated.

If an object has a contrast change only in the Y direction, as in thecase of object 520 in FIG. 5B, the image shift amount is calculatedusing the seventh image and the eighth image. Since the image shift isgenerated along the Y axis in this case, the distance to the object canbe accurately calculated.

However, the axis along which the contrast change is generated isunknown in advance. Therefore, in Embodiment 3, a plurality ofcombinations are generated for the images to be combined, the focusingdistance and the distance confidence are calculated for eachcombination, and the focusing distance of which confidence is highest isselected. As a result, the focusing distance can be accuratelycalculated.

FIG. 9A is a processing flow chart performed by the distance calculationunit 102 according to Embodiment 3.

Step S101 and step S102 are the same as step S1 and step S2 inEmbodiment 1, except that the number of processing target images is nottwo but four (the first image to the fourth image).

In step S106, a composite image is generated using four images. Inconcrete terms, the following four images are generated.

(1) The fifth image is generated by averaging the pixel values of thepixels located in the corresponding positions of the first image and thefourth image.(2) The sixth image is generated by averaging the pixel values of thepixels located in the corresponding positions of the second image andthe third image.(3) The seventh image is generated by averaging the pixel values of thepixels located in the corresponding positions of the first image and thesecond image.(4) The eighth image is generated by averaging the pixel values of thepixels located in the corresponding positions of the third image and thefourth image.

Step S103 is the same as step S3 in Embodiment 1, except that twopatterns of distance images are generated.

In concrete terms, a first distance image is generated using the fifthimage and the sixth image, and a second distance image is generatedusing the seventh image and the eighth image. The procedure in FIG. 3Bcan be used to generate the distance images.

First, to generate the first distance image, the fifth image is selectedas the standard image, and the sixth image is selected as the referenceimage. Then, to generate the second distance image, the seventh image isselected as the standard image, and the eighth image is selected as thereference image.

The baseline length for converting the image shift amount, which wascalculated based on the fifth image and the sixth image, into thedefocus amount is the distance between the fifth center of gravityposition 851 and the sixth center of gravity position 861. The baselinelength for converting the image shift amount, which was calculated basedon the seventh image and the eighth image, into the defocus amount isthe distance between the seventh center of gravity position 871 and theeighth center of gravity position 881.

Step S104 is a step of calculating the distance confidence. In thisstep, a first confidence image is calculated based on the fifth image,and furthermore, a second confidence image is calculated based on theseventh image. The procedure shown in FIG. 3C can be used to calculatethe confidence images.

Step S107 is a step of integrating the first distance image and thesecond distance image to generate a single distance image (integrateddistance image). In this step, a weighted mean of the first distanceimage and the second distance image is determined using the firstconfidence image and the second confidence image. In other words, thefirst confidence image and the second confidence image are compared, andthe first distance image and the second distance image are averaged suchthat the ratio of the distance having a higher confidence becomeshigher, whereby the integrated distance image is acquired.

As described above, according to Embodiment 3, the plurality of distanceimages and the confidence images are calculated by combining a pluralityof images, and a weighted integrated distance image is calculated sothat the confidence is increased. Thereby, the calculation accuracy ofthe focusing distance can be further improved.

In Embodiment 3, a plurality of distance images are calculated andweighted based on the confidence images before integrating the distanceimages, but the confidence images may be calculated first, and thedistance image may be calculated based on the calculation result.

FIG. 9B is a modification of the processing flow chart.

In this modification, a confidence image is calculated for eachcombination of the images of a composite image in step S104, and thedistance image is generated using the combination of the images of acomposite image of which confidence is the highest in step S103. Bycalculating the confidence images for each combination of the images ofa composite image in advance, the computation amount in step S103 can bereduced.

Further, in Embodiment 3, a case of generating the image shift along theX axis and a case of generating the image shift along the Y axis bychanging the combination of images of a composite image were described,but the direction of the image shift need not be a direction parallelwith the X axis or the Y axis. For example, a combination of the firstimage and the third image, or a combination of the second image and thefourth image, may be used in order to generate an image shift along anaxis that is 45° from the X axis. In any case, an axis, parallel withthe axis along which the image shift is generated, is defined as thefirst axis, and an axis perpendicular to the first axis is defined asthe second axis.

If the image shift can be generated by using the first image and thesecond image alone, then a composite image need not be generated.

Embodiment 4

In Embodiment 4, the method of calculating the distance confidence isdifferent from that in Embodiment 1.

FIG. 10A is a processing flow chart performed by the distancecalculation unit 102 according to Embodiment 4, and FIG. 10B is a flowchart depicting the processing executed in step S8 in detail.

A description of step S1 to step S3 and step S41 and step S42, which arethe same as Embodiment 1, will be omitted.

Step S81 is a step of estimating the noise amount.

In this step, the noise amount, that is generated in the image sensor101 when an object having a uniform brightness distribution isphotographed by the digital camera 100, is estimated in advance. Inconcrete terms, the noise generated in the image sensor is expressedusing an approximate expression, and the distribution of the noise onthe image detector plane is generated.

The noise generated in the image sensor 101 are: read noise, light shotnoise, dark current shot noise, and the like. Here, the noise amount isapproximated using Expression (6). Thereby, a noise amount estimationvalue (N(x,y)) at a pixel position (x,y) can be acquired.

ISO denotes an ISO sensitivity when an object is photographed by thedigital camera 100, and I(x,y) denotes a pixel value of the first imageat the pixel position (x,y). A and B denote approximate parameters.

[Math. 6]

N(x,y)=ISO·√{square root over (A ²+(B−I(x,y))²)}  Expression (6)

FIG. 11A is a graph showing an example of the noise amount estimationresult given by Expression (5). The abscissa indicates the pixel value,and the ordinate indicates the estimated noise amount.

In this example, by using this approximate expression, the graph becomesclose to the linear function when the pixel value is large, and becomesclose to the approximate parameter A, deviating from the linearfunction, when the pixel value is small. This is because the influenceof the light shot noise becomes more dominant as the pixel valueincreases, and the influence of read noise and the dark current shotnoise increases as the pixel value decreases.

If an approximate expression considering the noise characteristicgenerated in the image sensor 101 is used as in this example, the noiseamount can be more accurately approximated. In Expression (6), theapproximate parameter A is a constant, but may be a variable related tothe exposure time at photographing. By this, the influence of the darkcurrent shot noise can be more accurately expressed.

Expression (7) may be used to estimate the noise amount more easily.Here, max(a,b) denotes a function that returns a greater value of a andb. FIG. 11B shows the noise amount estimation result when Expression (7)was used.

[Math. 7]

N(x,y)=ISO−max(A,B·I(x,y))  Expression (7)

In step S82, a signal-to-noise ratio of the image (image S/N ratio) iscalculated using the representative value of the contrast change amountwhich was calculated in step S42, and the noise amount estimation valuewhich was calculated in step S81, and the calculated S/N ratio is outputas the distance confidence. The image S/N ratio can be calculated usingthe ratio of the representative value of the contrast change amount andthe noise amount estimation value.

As described above, according to Embodiment 4, the image S/N ratio iscalculated after the noise generated in the imaging pixel isapproximated, and the calculated image S/N ratio is defined as thedistance confidence.

The noise amount included in an image increases when high sensitivityphotography is performed, or when a high brightness object isphotographed by a digital camera. If the distance confidence iscalculated based on the image S/N ratio, the influence of the noise canbe reduced and the distance confidence can be calculated accurately evenif the noise amount is high.

In this embodiment, the pixel value of the pixel located at the position(x,y) is used to estimate the noise amount at the position (x,y) in stepS82, but the noise amount may be estimated using an average pixel valueof the pixels included in the collation region. By using the averagepixel value in the collation region, the noise amount can be estimatedunder conditions close to those when the image shift amount iscalculated. Hence, the distance confidence can be calculated even moreaccurately.

The expression to approximate the noise amount must be changedappropriately according to the method of evaluating the contrast changeamount. For example, the approximate expression shown in Expression (6)can be suitably used for calculating the first value using the varianceof the pixel values, but if the standard deviation of the pixel valuesis used, it is preferable to use an approximate expression that is closeto the square root function.

Embodiment 5

In Embodiment 5, a distance image generated using the generatedconfidence image is corrected.

FIG. 12A is a processing flow chart performed by the distancecalculation unit 102 according to Embodiment 5, and FIG. 12B is a flowchart depicting the processing executed in step S9 in detail.

A description of step S1 to step S4, which are the same as Embodiment 1,will be omitted.

Step S9 is a step of correcting the calculated distance image based on adistance image showing the distribution of the focusing distance, and aconfidence image corresponding to this distance image.

In step S91, a first pixel position is set in on the distance image, anda reference region centering around the first pixel position is set.

In step S92, a corrected distance corresponding to the first pixelposition is calculated. In concrete terms, by determining the weightedmean for the focusing distances included in the reference region, set instep S91, using the distance confidence, the corrected distances iscalculated. Then, the corrected distances are calculated while movingthe first pixel position, whereby the corrected distance image isgenerated.

Expression (8) expresses a method of determining the weighted mean forthe distance image using the confidence image.

[Math.  8] $\begin{matrix}{{{Dc}\left( {x,y} \right)} = \frac{\sum\limits_{y = {yp}}^{yq}\; {\sum\limits_{x = {xp}}^{xq}\; {{{Conf}\left( {x,y} \right)} \cdot {{Dr}\left( {x,y} \right)}}}}{\sum\limits_{y = {yp}}^{yq}\; {\sum\limits_{x = {xp}}^{xq}\; {{Conf}\left( {x,y} \right)}}}} & {{Expression}\mspace{14mu} (8)}\end{matrix}$

Dr(x,y) denotes a focusing distance at a position (x,y) in the distanceimage, Dc(x,y) denotes a corrected focusing distance at the position(x,y), and Conf(x,y) denotes a distance confidence at the position(x,y).

In this way, according to Embodiment 5, a weighted mean is determinedusing the distance image and the confidence image, and the correcteddistance image is calculated. Thereby, smoothing becomes possible withincreasing the ratio in a region where a calculation error of thedistance image is small. As a result, the focusing distance with evenhigher accuracy can be acquired.

In this embodiment, the weighted mean is determined using thedistribution of the distance confidence as shown in Expression (8), butthe weighted mean need not always be determined. For example, theconfidence image may be divided into a region where confidence is highand a region where confidence is low, using a predetermined threshold,so that the mean values are calculated using only the distance imagecorresponding to the region where confidence is high, whereby thecorrected distance image is calculated.

Further, in this embodiment, the distance image is corrected using onlythe confidence image, but image information may also be used for thecorrection as shown in Expression (9), for example. Ic(x,y) inExpression (9) denotes image information at position (x,y). The imageinformation refers to, for example, hue, brightness (e.g., colordifference and brightness difference based on position (x,y)), orcontrast change amount (second value). The weighted mean may bedetermined such that the ratio decreases as the position distances fromthe first pixel position.

[Math.  9] $\begin{matrix}{{{Dc}\left( {x,y} \right)} = \frac{\sum\limits_{y = {yp}}^{yq}\; {\sum\limits_{x = {xp}}^{xq}\; {{{Ic}\left( {x,y} \right)} \cdot {{Conf}\left( {x,y} \right)} \cdot {{Dr}\left( {x,y} \right)}}}}{\sum\limits_{y = {yp}}^{yq}\; {\sum\limits_{x = {xp}}^{xq}\; {{{Ic}\left( {x,y} \right)} \cdot {{Conf}\left( {x,y} \right)}}}}} & {{Expression}\mspace{14mu} (9)}\end{matrix}$

Embodiment 6

In Embodiment 6, the method of calculating the distance confidence isdifferent from Embodiment 1.

FIG. 14A is a processing flow chart performed by the distancecalculation unit 102 according to Embodiment 6, and FIG. 14B is a flowchart depicting the processing executing in step S10 in detail. Adescription of step S1 to step S3 and step S81 and step S82, which arethe same as Embodiment 4, will be omitted.

Step 10 is a step of calculating the confidence image corresponding tothe focusing distance calculated in step S3 (confidence calculationstep).

In step S101, a value indicating the contrast change amount (contrastevaluation value) is calculated. In this embodiment, the contrastevaluation value is calculated using the correlation degree that wasused when the image shift amount is calculated in step S31. In concreteterms, the slope, calculated from the correlation degree of a point nearthe corresponding point having the highest correlation, is defined asthe contrast evaluation value. If SSD was used for evaluating thecorrelation degree in step S31, the slope can be calculated byapproximating the correlation degree of a point near the correspondingpoint using a quadratic function. In other words, the slope SLOPE can becalculated using the following Expression (10).

[Math. 10]

SLOPE=(½)·(S(−1)−2·S(0)+S(+1))  Expression (10)

In Expression (10), S denotes a correlation degree based on SSD. S(0) isthe correlation degree of a reference point which has the highestcorrelation, S(−1) and S(+1) are correlation degrees when the movingdistance from the reference point having the highest correlation is a −1pixel and a +1 pixel respectively.

Step S81 is a step of estimating the noise amount. Just like Embodiment4, the noise amount is estimated using either Expression (6) orExpression (7). In this embodiment, an example when SSD is used forevaluating the correlation degree in step S31 will be described. SinceSSD uses a sum of squares of the differences between pixel values as theevaluation value, the contrast evaluation value calculated in step S101has degree n=2. It can be approximated that the light shot noise is inproportion to a square root of the number of photons. Hence, if a pixelvalue is large, the noise amount is estimated by approximating the pixelvalue using a linear polynomial or by using an expression close to alinear polynomial.

In step S31, the contrast evaluation value may be calculated based onthe slope of the correlation degree using sum of absolute difference(SAD), which uses the sum of absolute values of the differences as theevaluation value, instead of using SSD. In this case, the contrastevaluation value has degree n=1. If a pixel value is large in such acase, it is preferable that the noise amount is estimated byapproximating the pixel value using a square root function, or by usingan expression close to a square root function. In other words, it ispreferable that the estimated noise amount estimation value N iscalculated using the following Expression (11) or Expression (12), wheren is a degree of the contrast evaluation value, and A and B areconstants. The value I(x,y) denotes image information at the position(x,y).

[Math.  11] $\begin{matrix}{{N\left( {x,y} \right)} = {{ISO} \cdot {\sqrt{A^{n} + \left( {B \cdot {I\left( {x,y} \right)}} \right)^{n}}\left\lbrack {{Math}.\mspace{14mu} 12} \right\rbrack}}} & {{Expression}\mspace{14mu} (11)} \\{{N\left( {x,y} \right)} = {{ISO} \cdot {\max \left( {A,\left( {B \cdot {I\left( {x,y} \right)}} \right)^{\frac{n}{2}}} \right)}}} & {{Expression}\mspace{14mu} (12)}\end{matrix}$

In this embodiment, the pixel values of the pixel at the position (x,y)is used to estimate the noise amount at the position (x,y) in step S81,but the noise amount may be estimated using the mean pixel values of thepixels included in the collation region. By using the mean pixel valuesin the collation region, the noise amount can be estimated underconditions close to those used when the image shift amount wascalculated. Therefore, the distance confidence can be calculated evenmore accurately.

In this embodiment, the distance confidence is calculated in step S82,based on the ratio of the contrast evaluation value calculated in stepS101 and the noise amount calculated in step S81. However, if an amountof noise generated in the imaging pixels is high, the contrastevaluation value calculated based on the slope of the correlation degreemay in some cases be incorrectly calculated in step S101. Therefore, ifthe noise amount calculated in step S81 exceeds a predeterminedthreshold, a predetermined value, indicating that the distanceconfidence is low, may be used as the distance confidence in step S82,without calculating the ratio. Thereby, a case of incorrectlydetermining low distance confidence as high distance confidence can beprevented.

In this way, according to Embodiment 6, the image S/N ratio iscalculated after approximating the noise generated in the imagingpixels, and the calculated image S/N ratio is used as the distanceconfidence. When high sensitivity photographing is performed or when ahigh brightness object is photographed using a digital camera, theamount of noise included in the image increases. By directly calculatingthe noise amount included in the imaging pixels from the photographedimage, the noise amount is estimated even more accurately. As a result,the distance confidence can be calculated even when the noise amount ishigh.

(Modification)

The description of the embodiments is merely an example to describe thepresent invention, and the present invention can be carried out byappropriately modifying or combining the embodiments within a scope thatdoes not depart from the spirit of the invention. For example, thepresent invention may be carried out as a distance calculation apparatusthat includes at least a part of the above mentioned processing, or maybe carried out as a distance calculation method. Further, the presentinvention may be carried out as a program that causes the distancecalculation apparatus to execute this control method. The abovementioned processings and units may be freely combined as long as notechnical inconsistency is generated.

In the description of the embodiments, the digital camera was shown asan example of an imaging apparatus equipped with the distancecalculation apparatus according to the present invention, but thepresent invention may be applied to other apparatuses. For example, thepresent invention may be applied to a digital distance measuringapparatus.

In the description of the embodiments, a digital camera that has oneimaging optical system and one image sensor was shown as an example, buttwo imaging optical systems and two image sensors may be included as inthe case of the digital camera 1300 shown in FIG. 15A.

In this example, a first imaging optical system 1320 a and a secondimaging optical system 1320 b are the photographing lenses of thedigital camera 1300, and have a function to form an image of the objecton an image sensor 1301 a and an image sensor 1301 b, which are imagedetector planes, respectively.

The first imaging optical system 1320 a is constituted by a plurality oflens groups and a diaphragm, and has an exit pupil 1330 a at a positionthat is distant from the image sensor 1301 a by a predetermineddistance. In the same manner, the second imaging optical system 1320 bis constituted by a plurality of lens groups and a diaphragm, and has anexit pupil 1330 b at a position that is distant from the image sensor1301 b by a predetermined distance.

FIG. 15B is a diagram depicting the exit pupil 1330 a of the firstimaging optical system 1320 a viewed from the intersection of theoptical axis 1340 a and the image sensor 1301 a, and the exit pupil 1330b of the second imaging optical system 1320 b viewed from theintersection of the optical axis 1340 b and the image sensor 1301 b.

The first pupil region 210 is included in the exit pupil 1330 a, and thesecond pupil region 220 is included in the exit pupil 1330 b. The centerof gravity position 211 of the first pupil region 210 (first center ofgravity position) and the center of gravity position 221 of the secondpupil region 220 (second center of gravity position) are decentered(shifted) along the first axis 200.

Just like the digital camera 1300, the focusing distance and thedistance confidence can be calculated in the imaging apparatus having aplurality of imaging optical systems and image sensors using the samemethod as the method shown in the above examples.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or an apparatus that reads out and executescomputer executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., an application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., a central processingunit (CPU), or a micro processing unit (MPU)) and may include a networkof separate computers or separate processors to read out and to executethe computer executable instructions. The computer executableinstructions may be provided to the computer, for example, from anetwork or the storage medium. The storage medium may include, forexample, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a storage of distributed computing systems, anoptical disk (such as a compact disc (CD), a digital versatile disc(DVD), or a Blu-ray Disc (BD)™), a flash memory device, a memory card,and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. A distance calculation apparatus configured tocalculate distance information based on a first image, which isgenerated based on luminous flux that passed through a first pupilregion, and a second image, which is generated based on luminous fluxthat passed through a second pupil region, a center of gravity of thefirst pupil region and a center of gravity of the second pupil regionbeing located at different positions along a first axis, the distancecalculation apparatus comprising: a distance calculation unit configuredto calculate the distance information by comparing a local region of thefirst image and a local region of the second image; and a confidencecalculation unit configured to calculate, for a plurality of times, afirst value, which indicates a contrast change amount along the firstaxis, in a second axis direction crossing the first axis, in the localregion of the first image, in the local region of the second image, orin a local region of a composite image of the first image and the secondimage, and to calculate confidence of the distance information based ona second value, which is a value representing the plurality of firstvalues.
 2. The distance calculation apparatus according to claim 1,wherein the first value is at least one of (i) a variance of pixelvalues of a pixel array along the first axis, (ii) a difference betweena maximum value and a minimum value of the pixel values of the pixelarray along the first axis, and (iii) a sum of absolute values or a meanof absolute values of values generated by differentiating, along thefirst axis, the pixel values of the pixel array along the first axis. 3.The distance calculation apparatus according to claim 1, wherein thesecond value is at least one of a sum, a mean, a median and a mode ofthe plurality of first values.
 4. The distance calculation apparatusaccording to claim 1, wherein the distance calculation unit setsparameters, which are used for calculating the distance information,based on the calculated confidence.
 5. The distance calculationapparatus according to claim 1, further comprising a correction unitconfigured to correct the distance information, which has beencalculated by the distance calculation unit, based on the calculatedconfidence.
 6. The distance calculation apparatus according to claim 1,further comprising a first imaging optical system, wherein the firstpupil region and the second pupil region are included in an exit pupilof the first imaging optical system.
 7. The distance calculationapparatus according to claim 1, further comprising a first imagingoptical system and a second imaging optical system, wherein the firstpupil region is included in an exit pupil of the first imaging opticalsystem, and the second pupil region is included in an exit pupil of thesecond imaging optical system.
 8. An imaging apparatus comprising: animaging optical system; an imaging unit configured to generate an imagebased on luminous flux that passed through the imaging optical system;and the distance calculation apparatus according to claim
 1. 9. Adistance calculation method executed by a distance calculation apparatusconfigured to calculate distance information based on a first image,which is generated based on luminous flux that passed through a firstpupil region, and a second image, which is generated based on luminousflux that passed through a second pupil region, a center of gravity ofthe first pupil region and a center of gravity of the second pupilregion being located at different positions along a first axis, thedistance calculation method comprising the steps of: calculating thedistance information by comparing a local region of the first image anda local region of the second image; and calculating, for a plurality oftimes, a first value, which indicates a contrast change amount along thefirst axis, in a second axis direction crossing the first axis, in thelocal region of the first image, in the local region of the second imageor in a local region of a composite image of the first image and thesecond image, and calculating confidence of the distance informationbased on a second value, which is a value representing the plurality offirst values.
 10. A non-transitory computer readable storage mediumstoring a program to cause a distance calculation apparatus, configuredto calculate distance information based on a first image, which isgenerated based on luminous flux that passed through a first pupilregion, and a second image, which is generated based on luminous fluxthat passed through a second pupil region, to execute the steps of:calculating the distance information by comparing a local region of thefirst image and a local region of the second image; and calculating, fora plurality of times, a first value, which indicates a contrast changeamount along the first axis, in a second axis direction crossing thefirst axis, in the local region of the first image, in the local regionof the second image or in a local region of a composite image of thefirst image and the second image, and calculating confidence of thedistance information based on a second value, which is a valuerepresenting the plurality of first values, wherein a center of gravityof the first pupil region and a center of gravity of the second pupilregion are located at different positions along a first axis.